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RESERVE  COMPONENT  UNIT 
EVA  I. HAT  1 ON  ANALYSIS 
(Cost-E ffect  iveness) 


1.  INTRODUCTION 

A.  General.  This  interim  report  is  the  first  of  two  on  the  study  Reserve 
Component  Unit  evaluation  Analysis  (Cost-Effectiveness)  under  Contract  Number 
DAAG39-75-C-0135.  It  is  duly  submitted  three  months  after  the  start  of  the 
contract,  with  the  second  report  projected  for  delivery  in  January  1976.  A 
draft  final  report  will  be  submitted  not  later  than  sixty  days  before  the 
contract  termination,  with  the  final  report  to  be  delivered  in  mid-October  1976. 

B.  Purpose.  The  purpose  of  this  report  is  to  present  in  current  detail 

the  study  plan  for  analysis  and  to  apprise  the  Contract  Officer's  Representative 
(COR)  and  the  Study  Advisory  Group  (SAG)  of  work  progress  to  date.  The  report 
formally  reviews  and  updates  the  information  piesented  to  the  COR  and  the  SAG 
in  a briefing  at  the  Pentagon  on  27  March  1975. 

C.  Organization  of  the  IT  port . The  report  comprises  five  sections. 

1.  This  introduction  is  the  first  section. 
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2.  The  second  section  presents  the  background,  list  the  objectives, 
outlines  the  scope,  and  generally  describes  the  overall  approach  of  the  study 
effort . 


3.  The  third  section  defines  the  unit  evaluation  options  and  discusses 
the  methodology  for  selecting  optima. 


4.  The  fourth  section  identifies  cost  and  effectiveness  data  elements, 
discusses  sources  and  availability  of  data,  outlines  plans  for  data  collection, 
and  discusses  methods  of  summarization  and  presentation. 


5.  The  fifth  section  summarizes  the  plan  of  the  study  effort  for  the 
next  eight  months. 


II.  STUDY  OVERVIEW 


\J 


A.  Background.  In  developing  and  validating  the  Army  Training  and 
Evaluation  Program  (ARTEP)  TRAPOC  and  I'ORSCOM  efforts  have  centered  primarily 
on  demonstrating  the  concept’s  feasibility,  improving  the  ARTEP  utility  as  a 
guide  for  training  and  evaluation  of  Army  units,  and  very  recently  costing  the 
ARTEP.  Efforts  have  not  been  made  specifically  to  define  the  most  suitable 
ways  of  implementing  the  unit  evaluation  portion  of  the  ARTEP.  'Questions  of 
control ler/evaluator  numbers  or  sources,  frequon.  y of  evaluations,  applicability 
to  different  type  unit  . aggressor  sources,  and  the  like  have  not  been 
systematically  studied.  It  i essenti  1 that  the  more  effective  options,  in 


this  sense,  be  identified  and  that  the  costs  of  the  alternative  options  be 
considered  to  aid  the  Army  in  selecting  the  economical  ones.  Accordingly,  the 
Reserve  Component  Unit  I. valuation  Analysis  is  a cost-effectiveness  study  of 
options  for  evaluating  units  using  ARIlP. 

B.  Objectives. 

1.  To  analyze  alternative  approaches  implementing  the  ARTRI’  in 
assessing  the  effectiveness  of  Reserve  Component  units. 

2.  To  identify  the  costs  of  each  alternative  approach  to  include  money, 
men,  ami  collateral  impact. 

3.  To  recommend  assessment  systems  (fi.quency  of  testing,  manner  of 
application)  from  among  those  considered  for  use  in  periodic  Reserve  Component 
evaluations  with  the  ARTHP. 

4.  To  identify  units  (.by  type,  deployment  objectives,  mission  to  be 
tested)  with  which  the  assessment  systems  should  be  used. 

C.  Scope. 

1.  To  a large  extent  the  background  and  the  objectives  effectively 
determine  the  scope  of  the  study.  These  foregoing  are  specific,  for  example, 
that  the  study  concerns  Reserve  Component  uiits  and  selected  options  for  the  | 
implementation  of  unit  evaluations  under  ARTRI*.  Also,  they  express  clearly 
that  cost-effectiveness  analysis  is  required. 


2.  Additional  guidelines  relative  to  ARTEP  to  be  considered  and  data 
to  be  sought  art'  implicit.  Six  ARTEP  (test  editions)  are  currently  available. 

Of  these,  four  will  be  used  by  nine  Reserve  Component  battalions  during  AT  75 
as  a part  of  a TRADOC/FORSCOM  validation  program.  The  current  ARTEP 
Evaluation/Validation  Schedule  is  presented  in  Table  1.  Through  coordination 
with  CATB,  FORSCOM,  NOB,  State  Adjutants  General  concerned,  and  personnel  of 
the  participating  Reserve  Component  units  and  the  controller/evaluator  groups 
the  nine  evaluations  will  provide  the  nucleus  of  the  study  data  requirements. 
Other  relevant  data  will  be  available  from  the  review  and  analysis  of  existing 
data  (such  as  48U-R  oi  1-R  reports,  OSI)  Reserve  Component  Study  reports,  and 
planning  factor  literature),  interviews  with  Active  Army  and  Reserve  Component 
personnel,  and  selected  controlled  field  observations  during  AT  75  and  AT  76. 
Where  and  when  practicable  the  use  of  these  data  in  conjunction  with  other 
available  ARTEP  will  be  considered. 

3.  The  concentration  on  Reserve  Component  units  is  not  as  resti Lctive 
as  first  may  appear.  Early  observations  in  the  process  of  fashioning 
implementation  options  showed  that  a wider  range  of  evaluation  options  is 
possible  for  Reserve  Component  units  than  for  Active  Army  units  and  that  the  set 
of  Reserve  Component  options  includes  Active  Army  options  as  a subset.  Thus, 
some  Reserve  Component  options  found  to  be  cost-effective  well  may  be  applicable 
to  Active  Army  units. 
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Table  I 


ARTPP  I A All.  ITON/VALIOATION  SClU.nULH 

ui  slum:  componunts 

(as  of  24  April  1975) 


MAJOR  UNIT 

IYPP.  UNIT/ARTPP 

r l mi-:  i 

PP R 100 

LOCATION 

RMKS 

*8 1st  Inf  Bde  (M) 

Mech  Inf  Bn  ( 1 - 1 <>  1 IN) 

10-25 

Jun 

75 

Yakima 

L2 

(WA-ARNG) 

AR  Bn  (1-303  AR) 

f 

1 

r" 

Jul 

75 

Yakima 

L2 

I'A  Bn  (OS  Sp)  (2-140  PA) 

22  -2(> 

Jun 

75 

Lew  i s 

L2 

**3-117  Inf  (M) 

Mi , u Inf  Bn  (3-1  17  IN) 

10-18 

Jun 

75 

Hood 

L3 

(TN-ARNG) 

**1-123  AR 

AR  Bn  (1-123  AR) 

23-20 

Mar 

75 

Hood 

COMP  L 

(kV-ARNO) 

No 

Tk  Gnry 

***t)7th  Inf  Bde  (M) 

Mech  Inf  Bn  (1-134  IN) 

18-21 

Aug 

75 

Cp  Ripley 

L3 

(NL-ARNG) 

MN 

•> 

- COs 

Mech  Inf  Bn  (2-134  IN) 

11-14 

Aug 

75 

Carson 

L3 

AR  Bn  (1-105  AR) 

28-31 

Jul 

Carson 

L3 

No 

Tk  Gnry 

PA  Bn  (1-liuS  PA) 

0-12 

Jun 

75 

Cp  Guernsey 

1.2 

WY 

PN  Co  (St>7  l!N  Co) 

18-21 

Aug 

75 

Cp  Ripley 

L3 

mn 


•Affiliated  with  9th  Inf  Hi v 
**2d  AH  roundout  unit 
***Af f i 1 iatcd  with  4th  Inf  I'iv  (M) 


L2 , 1„>  indicate  level  tested,  as  shown  in  the  ARTUP 
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4.  The  end  product  ot'  the  study  will  be  a recommended  set  of  cost- 
effective  options  for  implementing;  Reserve  Component  evaluations  under  ARThP. 

If  different  options  are  more  cost-effective  for  different  types  of  units, 
uni t -speci f ic  implementation  programs  will  be  described. 

1).  Approach.  The  work  to  be  performed  is  divided  into  three  phases 
corresponding  to  the  three  major  tasks  described  in  the  contract  statement  of 
work:  Collection  of  Data,  Analysis  of  Alternatives,  and  Development  of  a 
Recommended  Program.  In  effect  the  work  throughout  the  performance  period  is 
a methodology  development  effort.  Also,  tasks  within  the  phases  are  not 
necessari ly  sequent ial . 

1.  The  first  phase  consists  of  planning  actions  and  data  collection 
necessary  for  the  start  and  conduct  of  subsequent  analytical  efforts.  The 
planning  includes  the  identification  of  data  elements  needed  for  cost- 
effectiveness  analysis,  the  review  of  existing  relevant  data  bases,  the 
development  oi  data  collection  materials,  and  the  collection  of  data.  The 
identification  of  data  elements  involves  the  definition  of  implementation 
options,  the  identification  of  major  cost  elements,  and  the  definition  of 
evaluation  effectiveness.  Some  of  the  planning  and  most  of  the  data  activities 
current  1) , necessari ly  (and  understandably)  continue  in  process.  Work  that  has 
been  accomplished  to  date  is  appropriately  the  subject  of  sections  III  and  IV 
of  this  report.  It  is  noted  that  the  submission  of  this  report  and  the 
completion  of  Phase  1 work  were  not  planned  to  be  coincident. 


2.  The  second  phase  involves  the  definition  of  analysis  parameters 
and  procedures,  the  analysis  of  collected  data,  and  the  development  of  relative 
option  costs  and  effectiveness  measures  to  identify  the  candidate  ARTEP 
evaluation  options.  The  work,  in  fact,  adapts/develops  and  applies  cost- 
effectiveness  analysis  procedures.  However,  because  the  practical  adaptation/ 
development  of  these  procedures  inherently  is  dependent  upon  the  kinds, 
quantity,  and  quality  of  data  to  become  available  for  analysis  they  cannot  yet 
he  finally  formulated.  This  tautology  notwithstanding  the  initial  work  in  this 
area  is  discussed  as  a part  of  section  111  in  this  report. 

3.  The  net  synthesis  of  Phases  1 and  2 produces  a subset  of  options 
estimated  to  be  cost-effective.  The  subset,  a complete  description  of  its 
derivation,  and  a recommended  program  for  testing  the  several  candidate  options 
in  practice  during  the  latter  half  of  TV  7b  and  AT  7b  are  the  subjects  of  the 
second  interim  report. 

4.  The  third  phase  involves  the  final  development  of  recommended 
assessment  options,  to  include  the  planning  and  monitoring  of  a limited  field 
implementation  of  the  recommended  systems.  Clearly,  the  planning  and  monitoring 
require  close  coordination  with  FORSCOM  and  TRAllOC.  As  a part  of  monitoring  the 
work  entails  data  collection  and  analysis  to  verify  the  estimates  associated 

with  the  recommended  options  and,  where  applicable  and  feasible,  to  extend  results 


to  other  ARTEP.  For  data  collection  it  is  envisaged  that  materials  developed  and 


used  in  Phases  1 and  2 may  be  adapted  and  that  additional  materials  can  be 


\ 

\\ 


I 
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developed  as  necessary.  This  phase  and  the  study  end  with  the  preparation 
and  submission  of  a final  report  of  all  work  accomplished  in  the  phase,  all 
previous  project  activities  (i.e.,  relevant  portions  from  the  two  interim 
reports),  and  final  conclusions  and  recommendations. 

111.  Al. IT: RNAT 1 Vi  1 MP LKMKNTAT 1 ON  0'’T  1 ONS 

A.  Option  Variables.  As  briefed  to  the  SAG  on  27  March  1975  three 
variables  - evaluation  schedule,  controller/evaluator  source,  and  basis  of 
application  - were  selected  for  describing  alternative  options  for  implementing 
Reserve  Component  unit  evaluations  under  ARTKP.  Subsequently,  the  last  of 
these  was  dropped  from  consideration  as  a variable  (see  below)  and  three  other 
variables  were  identified. 

1.  For  the  evaluation  schedule  variable  three  frequencies  - annual, 
biennial,  and  triennial  - were  chosen  for  consideration.  It  was  recognized  at 
the  outset  that  in  terms  of  current  practice  more  frequent  evaluations,  such  as 
annual,  might  place  too  heavy  demands  against  available  resources.  Yet,  with 
the  considerations  of  scheduling  AT  periods  throughout  the  calendar  year, 
conducting  some  evaluations  during  inactive  duty  training  (IDT),  and  using 
Reserve  Component  personnel  as  controllers/evaluators,  options  based  on  annual 
evaluations  may  prove  cost-effective. 

2.  Controller/evaluator  groups  may  be  all  Active  Army,  all  Reserve 
Component,  or  selected  mixes  of  Active  Army  and  Reserve  Component  personnel. 
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Although  there  are  oft  heard  oppositions  to  having  Reserve  Component  personnel 


evaluate  Reserve  Component  units,  the  opposition  may  not  outweigh  potentially 
cogent  cost-effective  arguments.  (Of  course,  no  cost-effectiveness  argument 
could  successfully  militate  against  invalid  test  conditions  whatever  the 
invalidity  factor.) 

3.  Basis  of  application  is  the  unit  mission  (or  type  unit)  or 
deployment  objective.  Under  consideration  following  discussions  with  agencies 
listed  in  Table  2 and  upon  the  suggestion  of  a SAG  member  that  deployment 
objective  is  not  properly  an  option  variable  it  was  dropped  from  consideration 
as  such.  However,  consonant  with  one  of  the  study  objectives  it  will  be  used 
in  the  final  review  of  options  selected  on  the  basis  of  the  cost-effectiveness 
analysis.  (See  "Other  Information  or  Guidance"  in  Figure  1.) 

B.  Other  Option  Variables.  The  three  other  option  variables  referred  to 
above  are  source  of  aggressor,  organizational  level  tested,  and  test  configuration. 

1.  The  aggressor  force  may  be  all  Active  Army,  as  it  is  in  the  AT  7S 
ARTEP  validation  exercises,  or  all  Reserve  Component.  Similarly  as  in  the  use 
of  Reserve  Component  personnel  as  controllers/evaluators,  there  is  known 
opposition  to  using  Reserve  Component  units  as  aggressor  forces.  Part  of  the 
argument  concerns  the  desirability  of  having  a standardized  aggressor  capability. 
However,  on  its  face  such  capability  cannot  be  assured  even  with  Active  Army 
aggressor  units.  With  either  Active  Army  or  Reserve  Component  aggressor  forces 
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Tabic  2 


Coordinat ion 

Visits 

*FEB 

24 

PENTAGON 

DA  DCS  OPNS  5 PLANS 

1 'i 

FEB 

27 

FT  MONROE,  VA 

TRADOC  - DCSOPS 

MAR 

4 

ARLINGTON,  VA 

ARMY  RESEARCH  INSTITUI'E 

MAR 

6 

FT  BENNING,  GA 

CATB  UNIT  TNG  5 RESERVE  COMPONENT 

MAR 

7 

FT  MCPHERSON,  GA 

FORSCOM  - DCSOPS 

MAR 

12 

PENTAGON 

NGB  - DCSOT,  DCSCOM  5 DCSLOG 

MAR 

18 

NASHVILLE,  TN 

TN  AG  5 USPFO 

i 

MAR 

19 

COOKEVILLE,  TN 

3/117  INF 

MAR 

19 

FT  KNOX,  KY 

ARMY  READINESS  REGION  VI 

MAR 

20 

FRANKFORT,  KY 

KY  AG  5 USPFO 

\ 

MAR 

23-26 

FT  HOOD,  TX 

III  CORPS  G3  8 DRC , 1st  CAV  DIV,  2nd 
MASSTER,  1/123  AR 

\ 

1 \* 

MAR 

27-28 

FT  MCPHERSON,  GA 

FORSCOM  - DCSOPS  8 DCSCOM 

APR  4 
APR  9 
APR  16 
APR  24 
APR  25 


PENTAGON 
FT  MEADE , MD 
RICHMOND,  VA 
FT  MCPHERSON,  GA 
FT  BENNING,  GA 


OCAR  - DCSOT 

FIRST  ARMY  DCSOI  5 OFFICE  OF  TNG  EVAL 

80  TNG  DIV  5 MTC 

FORSCOM  - DCSOPS  $ DCSCOM 

CATB  UNIT  TNG  5 RESC  COMP  AND  INF  SCHOOL 


First  of  on  going  visits  to  COR 
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controller  activity  plays  a significant  part  in  assuring  the  validity  of  the 
test.  Beyond  this,  whatever  the  merits  of  the  opposition  arguments  counter 
arguments  support ing  increased  Reserve  Component  personnel  morale,  additional 
Reserve  Component  unit  training,  and  potential  improved  cost-effectiveness 
require  the  consideration  of  this  variable,  finally,  the  use  of  Reserve 
Component  units  as  aggressor  forces  permits  simultaneous  evaluations  of  two 
units  under  the  opposing  forces  concept  - a potentially  economic  technique. 

2.  The  organizational  level  tested  is  largely  self-explanatory. 
ARTliP  evaluations  ma>  he  administered  to  battalions  or  to  companies.  Although 
the  ARTliP  also  include  platoon  and  squad/crew/team  testing,  it  is  felt  that 
for  the  study  these  organizational  levels  are  properly  included  with  the 
battalion  and  company  evaluations. 

3.  lest  configuration  delineates  the  evaluated  unit  pure  from  the 
evaluated  unit  combined  (i.e.,  with  cross  attached  subordinate  units).  This 
delineation  clearly  is  necessary  not  only  to  facilitate  differentiation  of 
costs  hut  also  to  facilitate  differentiation  of  effectiveness. 

4.  In  a discussion  at  one  agency  it  was  suggested  that  evaluation 
site  is  a variable.  The  concern  is  twofold:  (1)  that  all  evaluation  sites 
are  not  readily  suitable  for  all  evaluations,  and  that  therefore  there  are 
potential  cost  differences,  and  (2)  that  suitable  sites  do  not  exist  in 
sufficient  number  for  ARTliP  evaluations  of  all  units,  and  that  therefore  the 
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needed  number  of  sites  must  be  determined  and  the  costs  associated  with 
bringing  additional  suitable  sites  into  being  must  be  attributable  to  ARTIil* 
evaluation  implementation.  1'he  merit  of  this  concern  is  not  denied.  However, 
the  concern  is  seen  first  as  a Reserve  Component  unit  AT  scheduling  problem, 
and  second  as  a problem  to  be  addressed  after  cost-effective  options  are 
determined  and  selected  for  implementation. 

C.  Alternative  Options.  The  five  opt  ion  variables  - controller/evaluator 
source,  aggressor  source,  evaluation  schedule,  organizational  level  tested, 
and  test  conf igurat ion  - give  rise  to  seventy-two  basic  alternative  implementation 
options  plus  a to-be-determined  multiple  of  twenty-four  composite  options. 

1.  The  seventy-two  options  derive  from  constructing  all  possible 
sets  of  the  five  variables'  descriptive  elements  (.named  in  paragraphs  A-1,2 
and  B-1,2,3  above)  - with  the  Reserve  Component  element  of  the  controller/ 
evaluator  source  variable  divided  into  two  sub-elements:  Maneuver  Training 
Command  and  Other  Reserve  Component  personnel  (such  as  IISAR  Schools  and  training 
divisions).  The  seventy-two  optioi  . are  listed  in  the  first  three  pages  of  Table  3, 
where  each  horizontal  line  describes  an  option. 

2.  The  to-be-determined  multiple  of  twenty-four  composite  options 
derives  from  constructing  all  possible  sets  of  the  last  four  variables' 
descriptive  elements  (named  in  paragraphs  A -2  and  N-1,2,3  above)  and  combining 
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the  resulting  twenty  lour  sets  with  each  mix  of  Active  Army  and  Reserve 
Component  personnel  selected  to  constitute  a different  control ler/ evaluator 
group.  I'or  example,  for  a battalion  evaluation,  a group  might  consist  of  the 
chief  evaluator  and  other  evaluators  down  to  company  level  from  the  Active  Army 
and  the  remaining  evaluators  for  platoons  and  squad  s/crews/ 1 earns  from  the  Reserve 
Components.  A general  set  of  twenty-four  options  to  he  associated  with  any 
such  mixed  controller/evaluator  group  is  shown  on  page  4 of  Table  3.  It  is 
anticipated  that  the  number  of  mixed  controller/evaluator  groups  to  be 
considered  in  the  study  will  be  small. 

D.  Prime  Candidate  Options.  The  prime  candidate  options  are  those  from 
among  the  alternative  options  (Table  3)  selected  to  be  recommended  as  a part 
of  the  program  of  Reserve  Component  evaluations  under  ARTliP  during  \T  70.  The 
select  ion  of  prime  candidate  options  involves  two  procedures:  (1)  by  screening, 
the  determination  of  sets  of  candidate  options,  and  (2)  hv  cost-effectiveness 
ana  I vs i s , t he  determination  of  prime  candidate  options. 

1.  Screening.  The  screening  process  is  accomplished  in  two  steps, 
first,  each  ARTliP  (.Test  P.ditionl,  in  turn,  is  reviewed  to  find  where  in  its 
detail  it  is  incompatible  with  any  of  the  alternative  options  in  Table  3.  Such 
incompatible  options  are  selected  out  immediately,  leaving  a set  of  entry  options 
for  each  ARTPP  reviewed.  Second,  each  set  of  entry  options  is  reviewed  and 
discussed  with  as  many  ARTliP  cognizant  agencies  (Table  2)  as  possible.  Also, 


to  the  extent  possible  each  set  of  entry  options  is  reviewed  with  controllers/ 


evaluators  and  Reserve  Component  personnel  involved  in  the  AT  75  ARTliP 
validation  program.  Based  on  the  expert  military  opinions  rendered  in  these 
reviews  unlikely  entry  options  are  dropped  from  further  consideration,  leaving 
a set  of  candidate  options  for  each  ARTliP  reviewed. 

J.  Cost -liffect iveness  Analysis.  The  cost-effectiveness  analysis  is 
based  on  a simple  model  wiiicn  permits  the  manipulation  of  candidate  options 
and  their  associated  costs  and  effectiveness  to  achieve  optimum  visable  levels 
of  cost-effectiveness  among  the  several  ARTliP  evaluation  implementation  options. 
The  model,  Figure  1,  functions  as  follows: 

a.  Costs  for  each  option  in  each  set  of  candidate  options  are 
aggregated  to  give  total  cost  (or  total  incremental  cost)  for  each  candidate 
opt i on . 

b.  liffect iveness  data  for  each  option  in  each  set  of  candidate 
options  are  combined  to  give  an  index  of  effectiveness  (or  index  of  marginal 
effectiveness)  for  each  candidate  option. 

c.  If  a maximum  acceptable  option  cost  is  provided  (or  can  be 
determined)  options  having  costs  in  excess  of  the  maximum  are  dropped  from 
further  cons ide rat  ion. 
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Figure  1.  ANALYSIS  LOGIC 
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Figure  1.  ANALYSIS  LOGIC  (continued) 


d.  If  a minimum  acceptable  option  index  of  effectiveness  is 
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provided  (.or  can  be  determined)  options  having  indexes  smaller  than  the 
minimum  are  dropped  from  fuither  consideration. 

e.  For  each  remaining  option,  in  turn,  in  each  set  of  candidate 
options  the  ratio  of  the  index  of  effectiveness  to  the  total  cost  is  formed 
and  calculated,  or  the  index  of  effectiveness  and  the  total  cost  are  used  as 
rectangular  coordinates  to  plot  a representative  point.  Figure  2 illustrates 
the  plotting.  The  points  1,  2,  3,  4,  and  7 normally  would  not  be  plotted  on 
the  bases  of  paragraphs  c and  d preceding. 

f.  For  each  ARTEP  the  ratios  associated  with  its  set  of  candidate 
options  are  ranked  from  largest  to  smallest,  and  relative  to  a provided  (or 
determined)  criterion  the  prime  candidate  options  are  identified  as  those  whose 
ratios  exceed  the  criterion.  Where  representative  points  are  plotted  the 
decision  maker  has  several  choices.  With  reference  to  Figure  2,  for  example, 
points  5 and  (>  are  dominated  respectively  by  points  8 and  9 - i.e.,  with  the 
same  effectiveness  point  8 has  lower  cost  than  point  5 and  for  the  same  cost 
point  9 provides  more  effectiveness  than  point  t>.  Thus,  point  9 would  be  the 
choice  if  highest  effectiveness  is  desired  and  point  8 would  be  the  choice  if 
lowest  cost  is  desired.  Point  10  represents  the  option  with  the  most 
effectiveness  per  unit  cost.  Among  the  candidate  options  8,  9,  and  10  it  is 

the  optimal  efficiency  candidate  option,  and  would  be  the  decision  maker's 
choice  if  he  desires  the  "best  buy". 
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g-  Diagnostic  type  effectiveness  and  other  cost  information,  where 
applicable,  are  used  to  reduce  the  list  of  prime  candidate  options. 


3.  Sensitivity  Analysis.  Because  option  effectiveness  is  to  be 
expressed  in  terms  of  an  index  developed  from  subjective  and  generally 
qualitative  information  (ref  section  IV  C)  and  because  cost  data  are  to  be 
projected  across  options  there  is  an  element  of  uncertainty  in  the  cost- 
effectiveness  analysis.  Although  this  problem  cannot  be  avoided  in  cost- 
effectiveness  analysis,  it  also  cannot  be  ignored.  By  means  of  sensitivity 
analysis  methods,  testing  the  index  of  effectiveness  to  total  i.ost  ratio  across  a 
range  of  effectiveness  values  and  a range  of  cost  values,  it  will  be  determined 
which  elements  of  effect iw.-ness  and  which  elements  of  cost  have  the  greatest 


effect  within  the  set  of  prime  candidate  options.  The  sensitivity  analysis  also 
may  identify  the  region  of  ratio  results  of  greatest  sensitivity.  However,  because 
there  are  a large  number  of  options,  each  with  its  own  set  of  variables,  the  cost- 
effectiveness  model  may  exhibit  a built-in  sensitivity  analysis  technique  which 
will  make  a separate  analysis  unnecessary.  Howsoever  the  sensitivity  analysis  is 
accomplished  it  is  vitally  important  to  understanding  any  implications  of  the 
selected  prime  candidate  options. 


IV.  COST/ EFFECTIVENESS  DATA 


A.  General.  Cost-effectiveness  analyses  in  general  and  this  cost- 
effectiveness  analysis  in  particular  require  large  amounts  of  information  and 
data.  In  the  area  of  costs,  although  the  Army  generates  and  reports  significant 
amounts  of  quantitative  information,  not  all  required  cost  data  for  this  study 


analysis  are  available  - in  fact,  or  in  useable  form.  In  the  area  of  effectiveness 


measures  much  less  of  the  required  information  is  available  so  that  most  must 
be  generated  and  collected  during  the  study.  In  what  follows,  as  a part  of  work 
accomplished  during  this  first  phase  of  the  study,  cost  and  effectiveness  data 
requirements  are  defined,  sources  of  existing  data  are  identified,  and  plans 
for  collecting  data  are  described. 

B.  Cost  Data. 

1.  Requirements.  Various  interested  headquarters  and  agencies  were 
visited  in  person  and  by  telephone  to  assist  witli  the  identification  of  data 
elements  necessary  for  the  study.  In  conferences  with  FORSCOM,  Combat  Arms 
Training  Board  (CATB) , and  III  Corps  the  costs  listed  in  Table  4 were  identified. 

a.  Investment  Cost.  The  elements  of  investment  cost  to  be 
considered  are  those  associated  witli  positioning  equipment  and  construction 
costs  for  renovation  of  Annual  Training  (AT)  sites  required  to  support  AKTEP 
evaluations.  Investment  cost  will  be  amortized  to  reduce  the  impact  on  the 
evaluation  cost  in  the  first  year. 

b.  Operational  Cost.  Operational  cost  will  be  considered  only  to 
the  extent  that  it  is  incremental  to  the  support  of  ARTEP  evaluations.  The 
operational  costs  are  funded  by  Operation  and  Maintenance  as  well  as  Military 
Personnel  appropriations  for  Army,  National  Guard  and  Army  Reserve.  Procurement 
of  Ammunition,  Army  appropriation  funds  will  be  considered  if  ammunition  is 
determined  to  be  incremental. 
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Table  4 


cost  hi.hmpnts 


Investment  Cost 

Equipment  positioned  to  support  evaluations 
Site  renovation 
Operational  Cost 

Personnel  required  for  evaluations  segregated  by: 

Evaluation  Headquarters  - the  planning  and  evaluation  staff  responsible  for 
the  preparation  for  conduct  of  the  exercise. 

hvaluators  - personnel  responsible  for  the  conduct  of  the  evaluation 
Aggressors  - personnel  responsible  for  the  role  of  portraying  the  enemy 
Support  Personnel  - other  personnel  required  e.g.,  range  personnel 
Travel  necessary  for  personnel  support 
Per  diem  for  personnel  necessary  to  conduct  eva'uations 
POl,  consumed  for  evaluations  segregated  by: 
i-valuat ion  Headquarters 
Kvaluutors 
Aggressors 
Support  Personnel 

Maintenance  costs  to  include  repair  parts 
Ammuni t ion 


Personnel  added  to  the  MTC  IDA 


i ■mm 
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c.  Costs  not  considered  in  this  study. 

(1)  Evaluator  training  costs,  because  such  training  is 
conducted  prior  to  the  exercise  regardless  of  the  type  evaluation. 

(2)  Self  service  supply  center  costs,  because  supplies  will 
be  consumed  in  the  preparation  for  any  type  evaluation. 

f 3 ) Medical  support  costs,  because  such  support  is  required 
for  any  evaluation. 

C 4 ) l* ay  of  personnel  costs  for  the  unit  being  evaluated,  because 
they  are  present  regardless  of  type  evaluation. 

2.  Sources. 

a.  Investment  Cost.  No  investment  cost  data  have  been  identified 
to  date  because  evaluations  have  thus  far  been  accomplished  on  Active  Army 
installations.  Evaluations  may  require  investment  cost  to  permit  their  conduct 
as  implementation  of  ARTEP  is  expanded.  The  source  of  data  would  be  the 
installation  DCSCOM  for  Army  installations  and  the  State  United  States  Property 
and  Fiscal  Office  (USl’FO)  for  the  National  Guard. 

b.  Operational  Cost.  It  has  been  found  that  budget  data  is 
available  for  the  cost  to  support  evaluation  of  Reserve  Component  (RC)  units 
on  active  army  installations;  however,  a detailed  actual  cost  feedback  system 


is  not  available.  Costs  associated  with  training  are  available  in  gross 
figures  but  they  arc  not  broken  out  by  battalion  or  field  exercise.  Only 
incremental  cost  in  support  of  evaluations  associated  with  ARTliP  recognized 
requirements  will  be  considered.  Those  costs  incurred  which  are  above  those 
requirements  established  by  the  ARTliP,  e.g.,  for  the  sake  of  realism  or 
because  of  relationships  developed  through  the  affiliation  or  roundout 
programs,  will  not  be  considered.  A list  of  visits  for  coordination,  assistance, 
and  to  plan  data  collection  is  contained  in  Table  2 on  page  10.  Telephonic 
communications  with  the  personnel  visited  will  be  maintained  as  a part  of  keeping 
in  touch  witli  a dynamic  ARTliP  validation  program. 

11)  A visit  to  111  Corps  determined  that  budget  figures  are 
available  in  t lie  Directorate  of  Reserve  Components  for  support  of  1/123  Armor 
of  Kentucky  and  3/117  Infantry  of  Tennessee.  Actual  cost,  less  POL,  for  1/123 
Armor  will  be  available  in  the  Directorate  of  Reserve  Components  by  dune.  The 
visit  identified  the  cost  factor  tables  used  by  III  Corps  and  Modern  Army  Selected 
Systems  Test  Evaluation  and  Review  Activity  (MASSTF.R)  . They  were  based  on 
investment  costs  for  engineer  type  equipment  valued  up  to  $50,000.  These  costs 
were  extrapolated  to  $750,000  for  items  of  the  Army  equipment  known  to  cost 
more  than  $50,000.  Ill  Corps  charges  80".  of  the  factor  cost  because  the 
National  Guard  provides  the  POL  for  the  equipment  supplied  for  their  use. 

MASSTF.R  has  a data  base  for  the  tests  it  conducts  citing  all  equipment  used; 
however,  the  rate  factors  used  to  determine  cost  for  equipment  usage  arc  suspect. 
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(2)  The  1st  Cavalry  Division  lias  a man  performing  duty  as  a 


financial  manager  in  the  G3  office.  •lie  data  processing  system  used  to  control 
expenses  captures  only  about  (>0o  of  the  costs  he  is  responsible  to  control. 

The  factors  he  uses  for  costing  field  exercises,  based  on  a field  Operations 
Cost  Agency  document,  were  reviewed.  Controller  of  the  Army  advised  that  l-'OCA 
had  gone  out  of  business  and  their  data  was  obsolete. 

(3)  A visit  to  the  Kentucky  State  Adjutant  General  (AG)  and 
IISPFO  in  Frankfort  determined  that  POL  consumption  data  will  be  available  for 
both  1/123  and  2/123  Armor.  An  interesting  test  situation  existed  at 
Fort  Hood  in  that  1/123  Armor  was  evaluated  against  ARTHP  while  2/123  Armor 
was  evaluated  against  the  ATT.  It  may  he  possible  to  draw  conclusion  from 
a comparison  of  the  costs  in  this  case.  The  1/123  Armor  did  not  fire  the 
tank  tables.  Tne  ammunition  requirement  was  not  to  exceed  the  common 
table  of  allowance  (CTA)  limits. 

(1)  The  visit  to  the  Tennessee  State  Adjutant  General  and 
IISPFO  in  Nashville  developed  data  for  the  3/117  Infantry  for  Army  Training 
Test  (ATT)  in  AT  7-1  and  assurances  were  given  that  cost  data  would  be 
accumulated  for  AT  75  which  is  scheduled  for  dune.  The  3/117  Infantry  considers 
the  CTA  ammunition  allowance  adequate,  so  no  additional  cost  is  expected 
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for  ammunition. 
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(5)  FOKSCOM  DCSCOM  lias  AIT  cost  data  in  its  data  hank  but  it 
docs  not  include  data  for  the  National  Guard . Coordination  with  the  State 
AGs  and  USPFOs  became  more  important  since  all  lit!  units  utilizing  the  ARTHP 
in  AT  75  are  National  Guard.  It  was  confirmed  that  cost  factors  used  by  , 

111  Corps  for  costing  support  of  KC  units  and  MASSTLiR  for  various  tests  should 
not  be  used  because  of  the  data  base  and  the  extrapolation  used  to  develop 
the  tables. 

(())  CATB  is  developing  a cost  model  for  the  A RTF  I’  (training 
and  evaluation)  for  active  Army  units.  They  propose  to  collect  ARTF.P 
associated  costs  (POL,  Ammunition,  Maintenance  to  include  spare  parts,  and 
Supplies)  for  one  year.  Data  generated  may  be  applicable  to  the  RC  cost- 
effectiveness  study.  The  CATB  model  does  not  consider  the  personnel  required 
(evaluation  headquarters,  evaluators  and  aggressors)  to  support  the  evaluation. 

Liaison  will  be  maintained  with  CATB  for  applicability  to  the  RC  cost- 
effectiveness  study. 

(7)  Fort  Carson,  Colorado  and  the  4th  Infantry  Division  (M) 

DCSCOM,  in  recognition  of  the  fact  that  unit  training  activity  costs  do  not 
exist  at  division  level,  developed  a plan  to  obtain  actual  cost  data  on  POL, 
maintenance  and  repair  parts  in  field  exercises.  They  propose  to  develop 
cost  factors  for  operating  costs  (POL  and  repair  parts)  for  equipment.  While 
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the  ability  to  cost  training  activities  will  be  a valuable  tool  in  managing  j 

limited  training  funds,  the  adequacy  of  this  data  for  possible  utilization 
in  the  RC  cost  effectiveness  has  not  been  determined. 

(8)  Fort  Lewis,  Washington  and  the  9th  Infantry  Division 
have  not  yet  been  visited  to  determine  the  adequacy  of  data  on  RC  unit  support 
or  any  efforts  to  develop  the  cost  of  training  activities.  The  9th  Infantry 
Division  costs  for  ARTliP  validation,  as  reported  to  CATB  in  an  after  action 
report,  appear  to  he  excessive  and  require  clarification. 

(9)  FORSCOM  has  developed  an  effort  to  capture  POL,  maintenance 
and  spare  parts  cost  data  at  the  active  duty  division  installations.  The 
purpose  of  this  effort  is  to  assist  in  the  development  of  factors  for  the 
operation  and  maintenance  of  equipment  for  costing  RC  unit  support.  Collection 
is  planned  to  begin  on  July  1st.  FORSCOM  is  reviewing  the  Leavenworth  Model 

for  costing  a battalion  day  in  the  field  which  would  require  collection  of 
the  same  operating  costs  as  those  required  for  the  costing  support  of  RC  units. 

FORSCOM  also  plans  controlled  tests  to  capture  cost  data  for  both  ORTT  and  ARTF.P 
evaluations  t . . r battalions  (Armor,  Infantry,  and  Field  Artillery  as  a minimum) 
during  FY  75.  These  efforts  will  be  monitored  to  determine  applicability  to 
the  RC  cost-effectiveness  study. 
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^10)  The  Army  Readiness  Region  (ARR)  III  MTC  in  Richmond, 
Virginia  has  data  on  manduy  support  being  supplied  to  present  RC  ATT  evaluations. 
The  Maneuver  Training  Command  (MTC)  administrator  advised  he  felt  the  MIC 
would  welcome  the  opportunity  to  provide  evaluators  for  the  ARTEP  evaluations 
as  they  replace  the  ATI's  for  RC  units.  The  MTC  could  be  a repository  of 
evaluation  expertise  and  could  add  a measure  of  standardization  to  evaluations 
beyond  the  capabilities  of  an  Ad  Hoc  type  evaluation  group. 

(11)  Office  Chief  of  Army  Reserve  (OCAR)  has  a FORSCOM 
proposed  TDA  for  a type  MTC  recommending  an  increase  from  315  to  350  personnel. 
DAMO  has  recommended  the  approval  of  evaluators  be  made  tentative  pending 
completion  of  the  RC  cost-effectiveness  study  since  it  will  consider 
utilization  of  MTC  personnel  as  evaluators.  OCAR  advises  they  will  furnish 
cost  data  associated  with  the  increase  in  costs. 


3.  Data  Collection  Plan.  Existing  data  were  reviewed  and  collection 
accomplished  where  possible  on  tie  coordination  visits  listed  in  Table  2.  The 
bulk  of  the  data  required  for  this  study  remains  to  be  collected.  Visits  are 
planned  to  battalions  undergoing  their  ARTEP  evaluations  (Table  1),  the  staff 
at  the  installations  where  they  are  being  conducted,  the  State  Adjutants 
General  and  the  State  USPFOs  as  reflected  in  Table  5.  CATB  advised  that 
requirements  for  data  collection  may  be  included  in  the  after  action  report 


Tabic  5 

planned  data  collection  visits 


i t n i \ . n.i 
i t diyens,  ma 

FRAN k FORT , kV 
FT  kNOX , kV 
NASHVILLE,  TN 
TOPEKA,  kS 
LINCOI.N , NF. 
DENVER , CO 
FT  CARSON,  CO 
YAKIMA,  WA 
SFATTI.F. , WA 
FT  I.F.WIS,  WA 
FT  HOOD,  TX 


ARMY  RHAD1NFSS  RFC. ION  11 

ARMY  RFADINFSS  RFC  I ON  I 

STAFF  AG  fi  USPFO 

ARR  VI  !,  ARMOR  SCHOOL 

STAFF  AG  !I  USPFO 

STATE  AG  S USPFO 

STATE  AG  F,  USPFO 

STATE  AG  F,  USPFO 

Co,  G4 , COMPT,  DRC  F.  2/123  IN 

1/1  Cl  INFANTRY 

STATE  AG  5 USPFO 

Co.  G4 , COMPT,  DRC  F,  2/ 14c  FA 

G3 , Cl,  COMPT  F,  DRC 


instructions  CATB  will  issue  prior  to  each  evaluation.  To  the  extent 
possible,  efforts  will  be  made  to  adapt  existing  data  rather  than  impose 
new  data  collection  efforts.  Liaison  will  be  maintained  with  contacts  in 
the  field  to  keep  abreast  of  the  ARTEP  validations  and  data  collection  efforts. 

4.  Treatment.  The  cost  data  collected  will  be  treated  as  outlined 
in  the  methodology  described  in  paragraph  III  IK  The  analyses  and  processes 
to  be  utilized  are  subject  to  refinement  and  testing  throughout  the  span  of 
the  study  to  increase  the  confidence  levels  in  the  validity  of  the  results. 

C.  Effectiveness. 

1.  Definition.  Effectiveness  is  considered  to  be  function  of  the 
extent  to  which  ARTEP  evaluation  meets  its  stated  objectives  and  fulfills 
the  implicit  functions  of  any  evaluative  system,  namely,  to  provide  valid 

and  useful  feedback.  The  primary  concern  is  the  process  by  which  the  objectives 
are  met,  i.e.,  the  man  (evaluator!  - instrument  interface,  taking  the  instrument 
as  a given. 

2.  Common  ARTEP  Objectives. 

a.  "To  evaluate  the  ability  of  a [type]  battalion  to  serve  as 
a nucleus  of  a combined  arms  task  force  performing  specified  missions  under 
simulated  combat  conditions."  For  this  objective  effectiveness  determination 


is  concerned  primarily  with  the  accuracy  and  completeness  (together,  validity) 
of  the  information  rendered  through  conduct  of  the  evaluation. 

t>.  "To  provide  a guide  for  training  objectives  by  specifying 
minimum  standards  of  performance  for  combat-critical  missions  and  tasks." 

This  objective  relates  to  training  only  and  will  not  he  incorporated  in 
measures  of  evaluation  effectiveness. 

c.  "To  evaluate  the  efficiency  and  effectiveness  of  past 
training  of  all  echelons  of  the  battalion  from  crew/squad  through  battalion/task 
force."  Effectiveness  with  regard  to  this  objective  rests  upon  the  extent  to 
which  the  evaluation  yields  information  which  reflects  changes  in  unit  (or 
sub-unit/clemont)  performance  through  a test -train-retest  cycle. 

d.  "To  provide  an  assessment  of  future  training  needs.” 

Relative  to  this  objective  effectiveness  determination  is  concerned  with  the 
diagnostic  ability  of  the  evaluation  process,  i.e.,  its  ability  to  convey  the 
causes  of  mission  failure  or  unit/element  training  deficiencies  to  the  end  that 
evaluation  results  are  translatable  into  corrective  training  recommendations. 

3.  Data  Collection. 

a.  Background.  In  the  analysis  preparatory  to  undertaking  the 
current  study,  three  candidate  measures  of  effectiveness  (MOE)  were  defined: 
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adequacy  of  information,  user  acceptability,  and  promotion  of  proficiency  gains. 
The  two  former  MOE  are  retained  and  incorporated  with  the  above  effectiveness 
determinants  relative  to  ARTEP  objectives.  For  the  latter  indicator,  it  was 
generally  assumed  that  for  quantification  of  readiness  gains  either  training 
REDCON  levels  or  number  of  weeks  to  achieve  combat  proficiency  would  be  used. 
However,  a review  of  historical  data  to  obtain  such  readiness  indices  has  been 
suspended  for  three  basic  reasons.  First,  results  of  OSD  Tests  and  Affiliation 
Program  feedback  both  point  to  inflation  in  480-R  reports.  Second,  the  four- 
point  readiness  level  scale,  of  which  only  the  middle  two  are  commonly  assigned, 
represents  a scale  insensitive  to  all  but  substantial  changes  in  proficiency. 

Last,  and  most  cogent,  this  study  is  concerned  only  with  the  evaluation 
function  of  ARTEP,  while  proficiency  gains  represent  at  best  an  indirect  index 
of  evaluation  effectiveness  due  to  the  many  other  factors  which  impinge  upon 
training,  e.g.,  leadership,  personnel  turbulence,  time  and  resource  constraints. 

b.  Identification  of  Data  Elements.  Data  requirements  are 
summarized  in  Table  6. 

(1)  It  has  been  determined  that  all  data  needed  for  the 
conduct  of  effectiveness  analysis  will  be  obtained  from  on-going  and  future 
activities,  to  include  observation  of  ARTEP  evaluation  and  validation 
exercises,  surveys  and  in-depth  interviews  of  personnel  involved,  large-scale 
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Table  b 


EFFECTIVENESS  DATA  REQUIREMENTS 


EFFORT  * 


DATA  SOURCE 


PURPOSE 


Large  pool  of  AA  fi  RC  Officers 
(echelons  up  to  L)A  fi  NGB)  : vii 
mail -out  questionnaires 


To  develop  effectiveness  index 
relative  to  implementation 
options  for  use  in  co:.t- 
effectiveness  model 


CO  fi  S-3  of  all  tested  units 
fi  responsible  parties  in 
reporting  chain:  via 
structured  interviews/ 
follow-up  questionnaires 


To  poll  user  acceptability  5 
gain  judgment  of  information 
adequacy 


ARTF.P  Validation  results, 
after  action  reports, 

FORSCOM  Form  1-R,  ARTEP 
evaluators  via  questionnaires, 
FORSCOM  evaluators  via 
questionnaires,  fi  unit  training 
personnel  via  structured  inter- 


AT  75  ARTEP  evaluation  results 
AT  7b  ARTEP  evaluation  results 


To  judge  accuracy  § complete- 
ness of  feedback  from  ARTEP 
evaluation  using  independent 
data  source 


To  assess  the  efficiency  of 
ARTEP  evaluation  to  produce 
feedback  relative  to  training 
effectiveness 


♦Numbers  correspond  to  efforts  outlined  in  paragraph  4. 


mailings  of  questionnaires  in  two-phases,  and  systematic  review  and  analysis 
of  ARTEP  results  and  evaluator  comments. 

(2)  Coordination  visits  made  to  date  and  those  anticipated 
in  the  near  future  are  depicted  in  Tables  2 and  5,  (pp  10,  32)  respectively. 

(3)  During  coordination  visits  the  study  and  how  it  relates 
to  current  Army  efforts  were  discussed  at  length.  Visited  agencies  are 
cooperating  to  furnish  pertinent  data  as  they  are  obtained.  Additional 
recommendations  and  suggestions  for  ARTEP  implementation  options  to  be 
considered  were  solicited.  During  coordination  visits  the  various  effectiveness 
elements  and  options  also  were  discussed  with  RC  and  Active  Army  personnel. 

(4)  Reports  of  the  on-going  ARTEP  validation  exercises  are 
a basic  source  of  effectiveness  data.  These  reports  teamed  with  in-depth 
discussions  with  TRADOC,  F0RSC0M,  and  CATB  personnel  plus  evaluators  and  others 
involved  with  ARTEP  and  ARTEP  implementation  constitute  the  core  of  the 
effectiveness  data  sources. 

4.  Approach.  There  will  be  four  separate  but  related  efforts  involved 
in  the  analysis  of  ARTEP  implementation  effectiveness. 
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II 


a.  The  first  and  major  effort  is  directed  toward  predicting  the 
relative  ability  of  implementation  options  to  render  accurate  and  usatie 
information  t relating  to  ARTEP  objectives  a,  c and  d) . This  determination 
will  be  derived  through  scaling  [psychometric  projection]  of  relative 
evaluation  effectiveness  of  various  implementation  schemes  based  on  expert 
military  judgment.  This  effort  can  be  expected  to  reflect  projections  of 
user  acceptability  as  well.  The  detailed  methodology  for  this  effort  is 
presented  in  the  following  section  (para  5) . 

b.  The  second  effort  will  attempt  to  ascertain  user  acceptability 
at  all  levels  of  responsibility  and  interest.  This  indicator  of  effectiveness 
will  be  gained  through  post-exercise  interviews  and  follow-up  questionnaires, 
as  well  as  through  review  of  validation  after-action  reports.  Coordination 
and  cooperation  is  anticipated  with  CATB  and  FOKSCOM  for  the  collection  of 
mutually  supportive  documentation  during  and  immediately  following  validation 
exercises.  This  effort  is  concerned  with  user  acceptability  of  the  evaluation 
system,  as  well  as  the  usefulness  of  information  derived.  The  thrust  of  this 
effort  will  be  to  determine  if  responsible  parties  at  each  progressive  echelon 
feel  that  they  have  the  type  and  quantity  of  information  they  require  to  make 
valid  decisions  and  to  meet  their  directive  and  reporting  functions.  This  effort 
relates  to  ARTEP  objective  a,  c and  d)  . 


The  third  effort  consists  of  three  steps. 


c . 


(.1)  The  first  step  is  aimed  at  an  independent  assessment 
of  the  accuracy  and  sufficiency  of  feedback  data  and  will  constitute  a 
systematic  review  of  ARTliP  results.  This  initial  step  will  consist  of  an 
item  comparison  of  ARTliP  results  (deficiencies  noted)  with  the  critique  of 
training  readiness  rendered  by  t he  three-man  Active  Army  evaluation  team 
present  during  the  same  exercise  (and  throughout  AT)  but  not  involved  in  the 
ARTliP  implementation  or  evaluation.  While  it  is  acknowledged  that  reporting 
requirements  of  the  team  differ  from  that  of  ARTliP  evaluators,  both  are 
assessing  training  proficiency  and  interested  in  noting  weaknesses  w i thin  the 
evaluated  unit.  The  comparison  of  ARTliP  results  and  the  independent  evaluation 
by  the  three-man  team  will  look  at  the  relative  amounts  of  information 
generated,  the  separate  type  of  information  provided,  noted  deficiencies 
which  arc  absent  in  either  source,  and  essentially  the  degree  to  which  they 
represent  parallel  or  divergent  perspectives  of  a unit's  level  of  training. 

(2)  The  second  step  of  this  effort  will  seek  verification 
of  ARTliP  results  and  the  readiness  evaluation  team  data  (in  terms  of  overall 
evaluation  and  specific  deficiencies  noted)  from  evaluated  unit's  personnel 
(especially  CO  and  S-3)  . This  step  will  be  accomplished  by  follow-up 
questionnaire  or  interview,  tailored  to  each  unit,  after  all  exercise  reports 


have  been  submit  ted  and  commanders  and  training  personnel  have  had  adequate 
time  to  review  them.  The  purpose  of  this  step  is  to  achieve  an  indication 
of  relative  perceived  accuracy  of  the  two  sources  of  evaluative  information. 

(.3)  l-'inully  we  will  want  to  learn  from  battalion  personnel 
if  there  are  any  known  areas  of  deficiency  not  detected  by  either  evaluation 
procedure.  The  integration  of  the  three  above  steps  should  yield  a good, 
qualitative  indication  of  the  accuracy  and  completeness  of  feedback  data  from 
the  ARTliP  evaluation.  (.This  effort  relates  to  ARTliP  objective  a.) 

d.  The  proposed  fourth  analytical  effort  is  concerned  with  the 
effectiveness  of  the  ARTliP  evaluation  in  providing  feedback  relative  to 
efficiency  of  training  conducted  during  IDT.  This  task  will  entail  comparison 
of  same-unit  ARTliP  results  from  AT  73  and  AT  70.  It  is  recognized  that  certain 
test  units  will  need  to  be  designated  to  undergo  re-evaluation  out  of  phase  for 
this  task  to  be  undertaken.  Also,  the  evaluations  should  be  as  comprehensive 
as  possible  or  at  least  include  the  same  types  of  missions.  Comparison  of  two 
year  test  results  will  yield  three  categories:  areas  of  deterioration,  areas 
of  no  change,  and  areas  of  improvement.  These  three  categories  would  then  hi 
corresponded  to  instructional  elements  of  the  unit's  TY  7(>  training  program. 
Analysis  will  have  also  to  consider  the  rate  and  levels  of  personnel  turbulence 
incurred  by  the  unit  during  TY  70.  In  total  this  technique  will  point  up  types 
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ot"  training  which  were  successful,  those  that  were  less  efficient,  and  others 
which  were  lacking.  It  will  also  be  possible  to  determine  through  this  effort 
the  extent  to  which  the  TY  7o  training  program  emanated  from  results  of  ARTFP 
AT  75.  (This  approach  relates  to  ARTFP  objective  c.) 

5.  Field  experimentation.  In  addition  to  the  four  efforts  outlined 
in  the  approach  above  (para  1),  experimental  comparison  of  prime  candidate 
implementation  options  may  be  possible  through  either  of  two  methods.  If 
prime  candidate  options  are  field  tested  during  AT  7t>,  comparison  of  AT 
evaluation  effectiveness  ratings  for  two  or  more  like  units  tested  under 
different  options  should  offer  direct  verification  of  rankings  achieved  in 
the  first  major  effort  described  below  (para  5).  The  other  access  to 
experimental  comparison  could  be  achieved  in  conjunction  with  effort  number 
four  (para  4 d) . Thi s latter  comparison  would  be  possible  for  same  units 
evaluated  under  different  options  during  AT  75  and  AT  7t>. 

t>.  Method.  Following  is  a discussion  of  the  methodology  anticipated 
in  the  execution  of  the  first  effort  to  achieve  a quantitative  scaling  of 
ARTIiP  implementation  options.  The  second,  third,  and  fourth  efforts  described 
above  will  yield,  in  general,  more  qualitative  and  diagnostic  information  and 
will  be  used  to  modify  the  results  of  the  first  effort.  It  should  further  be 
noted  that  efforts  2-4  will  yield  indications  of  relative  effectiveness  only 
for  options  which  have  actually  been  field  tested,  but  as  conceived,  will 
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not  generate  data  pertinent  to  untested  implementation  options.  The 


methodology  comprises  these  five  steps: 


• Identify  option  variables 


• Assemble  option  ranking  packages  and  rating  packages  to  be  completed 
by  two  groups  of  military  judges 


• From  ratings  of  expert  military  judges,  develop  effectiveness  index 
for  each  option 

• From  rankings  of  option  subset  by  expert  military  judges,  develop 
option  ranks 


• Using  ranking  and  rating  results,  conduct  sensitivity  analysis  to 
assess  the  contribution  of  option  variables  to  option  effectiveness. 

a.  Initially  the  options,  arranged  in  subsets  due  to  their 
large  number,  will  be  presented  to  a large  sample  of  Reserve  Component  and 
Active  Army  officers  for  the  assignment  of  ratings  on  a 0-9  scale.  The 
number  nine  on  the  scale  will  lie  defined  as  belonging  to  an  "ideal"  evaluation 
system  and  :ero  will  represent  the  absence  of  evaluation.  Establishing  a zero 
reference  point  will  enable  ratings  to  be  treated  in  the  manner  of  a ratio. 

The  ideal  evaluation  system  will  be  described  as  being  capable  of  identifying 
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and  diagnosing  100%  of  all  actual  deficiencies  while  making  no  specious 
identification,  and  being  able  to  correspond  all  deficiencies  with  requisite 
corrective  training  compatible  with  a unit's  training  support  resources. 

Results  of  the  rating  exercise  will  be  normalized  and  converted  to  scale  values 
using  standard  statistical  procedures.  These  scale  values  will  be  the  basis  of 
the  effectiveness  index  to  be  used  in  the  cost-effectiveness  ratio. 

b.  Concurrently,  mixed  subsets  of  options  will  be  submitted  to  a 
second  group  of  military  judges  to  be  placed  in  rank  order  from  "most  effective" 
to  "least  effective."  Subset  ranks  will  be  integrated  to  achieve  an  overall 
ranking  of  the  total  set  of  options.  Sensitivity  analysis  can  be  performed 
on  the  rankings  to  determine  the  relative  importance  (weight)  of  the  five 
implementation  variables  to  evaluation  effectiveness . This  will  be  accomplished 
by  examining  the  distribution  of  descriptive  elements  of  each  variable  within 
the  rankings.  (Examples:  a cluster  among  the  high  ratings  indicates  a strong 
positive  influence;  even  distribution  throughout  indicates  little  or  no 
influence;  a cluster  among  low  rankings  indicates  strong  negative  influence.) 

The  rankings  will  also  be  used  to  establish  the  level  of  confidence  to  be 
attached  to  option  ratings.  This  is  to  be  done  through  an  examination  of 
correspondence  between  results  of  ranking  and  rating  exercises. 


Both  samples  of  judges  will  he  comprised  of  1/3  Active  Army, 
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1/3  IISAR,  and  1/3  ARNti.  The  emphasis  in  the  collection  sample,  as  in  the 
overall  study,  is  on  the  Reserve  Components.  For  purposes  of  analysis,  RC 
and  AA  respondents  will  he  treated  as  separate  populations  because  components 
of  evaluation  options  may  differ  in  effectiveness  between  the  Active  Army  and 
Reserves . 

d.  As  suggested  earlier,  the  validation  exercises  run  during 

CY  75  will  present  implementation  options  for  which  effectiveness  index  values 
can  he  generated.  Other  options  to  he  examined  can  then  he  considered  of 
incremental  or  decremental  effectiveness  relative  to  these  base  cases. 

e.  In  disseminating  the  rating  packages,  ARTEP  will  he  described 
as  the  context  for  t he  evaluation,  hut  direct  experience  with  ARTEP  or 
conceptual  understanding  is  not  considered  essential  for  making  the  judgments 
called  for  above,  effectiveness  ratings  for  different  versions  of  ARTEP  will 
he  handled  in  the  following  manner.  Options  for  implementation  of  the 
infantry  ARTF.P  will  he  rated  by  infantry  officers  only,  signal  ARTEP  by  signal 
officers,  armor  ARTEP  by  armor  officers,  and  so  forth.  For  variables  or  options 
relevant  to  all  ARTEPs,  responses  can  he  pooled.  For  those  that  are  relevant 
only  to  specific  versions,  they  will  he  analyzed  by  branch  (Example:  source 

of  aggressor  is  not  relevant  to  ARTF.P  11-35,  Signal  Bn).  Evaluators  who  have 
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participated  in  ARTliP  validation  exorcises  will  be  requested  to  complete  the 


same  rating  task-,  but  will  be  considered  a separate  population,  with  the 
intended  purpose  of  offering  an  objective  comparison  (i.e.  - a check  for  bias). 


V.  SUMMARY 


A.  The  foregoing  sections  present  an  overview  of  the  Reserve  Component 
Unit  evaluation  Analysis  (Cost-effectiveness)  study,  a discussion  and  list 
of  alternative  options  for  implementing  evaluations  under  ARTliP,  a method 
for  selecting  prime  candidate  options,  a list  of  cost  data  requirements  and 

an  associated  data  collection  plan,  a discussion  of  effectiveness  measurement, 
a plan  for  obtaining  effectiveness  information,  and  an  overall  report  of  work 
done  to  date. 

B.  The  work  done  to  date,  including  the  preparation  of  the  first  interim 
report,  is  a part  of  Phase  1 in  this  three  phase  study.  Phase  1 work  now 
continues.  It  entails  the  designing  and  developing  of  interview  forms,  surveys, 
data  recording  forms,  and  tabulation  worksheets  - coordinated  with  the  Army 
ARTliP  evaluation  team  as  necessary;  coordinated  receipt  of  AT  75  Reserve 
Component  unit  evaluation  after  action  reports  and  other  validation  information 
as  it  becomes  available;  coordinated  collection  of  cost  data;  coordination 
visits  and  re-visits  to  cognizant  military  agencies;  evaluation  exercise 
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observation  and  information  gathering  visits  to  AT  sites;  visits  to  unit 
headquarters  to  conduct  interviews  and  to  collect  cost  data;  conduct  of 
surveys  by  mail. 

C.  Phase  2 work,  some  of  which  overlaps  with  Phase  1,  commences  with  the 
analysis  of  cost  and  effectiveness  data  on  an  option-by-option  basis,  with 
differentiation  by  unit  type,  specifically  for  those  options  employed  in  the 
validation  exercises  during  AT  75;  includes  the  projection  of  cost  and 
effectiveness  data  to  other  options  as  pertinent,  with  associated  analysis; 
proceeds  with  the  cost-effectiveness  analysis  method,  shown  in  Figure  1, 

(page  19)  to  select  prime  candidate  implementation  options;  and  culminates 
in  January  197b  with  the  submission  of  the  second  interim  report  presenting 
recommendations  for  testing  the  prime  candidate  implementation  options  in 
Reserve  Component  Unit  evaluations  under  ARTUP  during  the  latter  ha  1 f of 
TY  7b  and  AT  7b. 
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